Introduction
- TMLE is a general algorithm for the construction of double robust, semiparametric, efficient substitution estimators. TMLE allows for data-adaptive estimation while obtaining valid statistical inference.
- Although TMLE implemtation uses the G-computation estimand (G-formula). Briefly, the TMLE algorithm uses information in the estimated exposure mechanism P(A|W) to update the initial estimator of the conditional mean E\(_{0}\)(Y|A,W).
- The targeted estimates are then substituted into the parameter mapping. The updating step achieves a targeted bias reduction for the parameter of interest \(\psi(P_{0})\) (the true target parameter) and serves to solve the efficient score equation. As a result, TMLE is a double robust estimator.
- TMLE it will be consistent for \(\psi(P_{0})\) is either the conditional expectation E\(_{0}\)(Y|A,W) or the exposure mechanism P\(_{0}\)(A|W) are estimated consistently. When both functions are consistently estimated, the TMLE will be efficient in that it achieves the lowest asymptotic variance among a large class of estimators. These asymptotic properties typically translate into lower bias and variance in finite samples.(Bühlmann et al., 2016)
- The advantages of TMLE have been repeatedly demonstrated in both simulation studies and applied analyses.(Laan and Rose, 2011)
- The procedure is available with standard software such as the tmle package in R (Gruber and Laan, 2011).
Causal assumptions
Under the counterfactual framework, we have to consider the following assumptions to consider the estimate of the ATE as causal:
CMI or Randomization
(\(Y_{0},Y_{1}\perp\)A|W) of the binary treatment effect (A) on the outcome (Y) given the set of observed covariates (W), where W = (W1, W2, W3, … , Wk).
Positivity
a ϵ A: P(A=a | W) > 0
P(A=1|W=w) > 0 and P(A=0| W = w) > 0 for each possible w.
Consistency or SUTVA:
The observed outcome value, under the observed treatment, is equal to the counterfactual outcome corresponding to the observed treatment for identical independent distributed (i.i.d.) variables.
TMLE flow chart
Source : Mark van der Laan and Sherri Rose. Targeted learning: causal inference for observational and experimental dataSpringer Series in Statistics, 2011 
Data generation
In R we create a function to generate the data with the input number of draws and the output the observed data (ObsData) plus the counterfactuals (Y1, Y0).
The observed data:
1. Y: mortality binary indicator (1 death, 0 alive) 2. A: binary treatment for emergency presentation at cancer diagnosis (1 EP, 0 NonEP)
3. W1: Gender (1 male; 0 female)
4. W2: Age at diagnosis (0 <65; 1 >=65)
4. W3: Cancer TNM classification (scale from 1 to 4)
5. W4: Comorbidities (scale from 1 to 5)
#install.packages("broom")
options(digits=3)
generateData <- function(n){
w1 <- rbinom(n, size=1, prob=0.5)
w2 <- rbinom(n, size=1, prob=0.65)
w3 <- round(runif(n, min=0, max=4), digits=3)
w4 <- round(runif(n, min=0, max=5), digits=3)
A <- rbinom(n, size=1, prob= plogis(-0.4 + 0.2*w2 + 0.15*w3 + 0.2*w4))
Y <- rbinom(n, size=1, prob= plogis(-1 + A -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
# counterfactual
Y.1 <- rbinom(n, size=1, prob= plogis(-1 + 1 -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
Y.0 <- rbinom(n, size=1, prob= plogis(-1 + 0 -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
# return data.frame
data.frame(w1, w2, w3, w4, A, Y, Y.1, Y.0)
}
set.seed(7858)
ObsData <- generateData(n=1000)
True_Psi <- mean(ObsData$Y.1-ObsData$Y.0);
cat(" True_Psi:", True_Psi)
True_Psi: 0.24
Bias_Psi <- lm(data=ObsData, Y~ A)
cat("\n")
cat("\n Naive_Biased_Psi:",summary(Bias_Psi)$coef[2, 1])
Naive_Biased_Psi: 0.21
Data visualization
# DT table = interactive
# install.packages("DT") # install DT first
library(DT)
datatable(head(ObsData, n = nrow(ObsData)), options = list(pageLength = 5, digits = 2))
TMLE implementation
1st step: E\(_{0}\)(Y|A,W)
Estimation of the initial probability of the outcome (Y) given the treatment (A) and the set of covariates (W), denoted as the \(Q_{0}\)(A,W). To estimate \(Q_{0}\)(A,W) we can use a standard logistic regression model:
\(logit[P(Y=1|A,W)]\,=\,\beta_{0}\,+\,\beta_{1}A\,+\,\beta_{2}^{T}\)W.
Therefore, we can estimate the initial probability (as follows: . (1) The predicted probability can be estimated using the Super Learner library implemented in the R package “Super-Learner”6 to include any terms that are functions of A or W (e.g., polynomial terms of A and W, as well as the interaction terms of A and W, can be considered). Consequently, for each subject, the predicted probabilities for both potential outcomes and can be estimated by setting A = 0 and A = 1 for everyone respectively: and,.
Thank you
Thank you for participating in this tutorial.
If you have updates or changes that you would like to make, please send me a pull request. Alternatively, if you have any questions, please e-mail me.
Miguel Angel Luque Fernandez
E-mail: miguel-angel.luque at lshtm.ac.uk
Twitter @WATZILEI
Session Info
devtools::session_info()
References
Bühlmann P, Drineas P, Laan M van der, Kane M. (2016). Handbook of big data. CRC Press.
Greenland S, Robins JM. (1986). Identifiability, exchangeability, and epidemiological confounding. International journal of epidemiology 15: 413–419.
Gruber S, Laan M van der. (2011). Tmle: An r package for targeted maximum likelihood estimation. UC Berkeley Division of Biostatistics Working Paper Series.
Laan M van der, Rose S. (2011). Targeted learning: Causal inference for observational and experimental data. Springer Series in Statistics.
LS0tCnRpdGxlOiAiVE1MRSBzdGVwIGJ5IHN0ZXAiCmF1dGhvcjogJ0J5OiBNaWd1ZWwgQW5nZWwgTHVxdWUgRmVybmFuZGV6LCBtaWd1ZWwtYW5nZWwubHVxdWVAbHNodG0uYWMudWsnCmRhdGU6ICJPY3RvYmVyIDE1dGgsIDIwMTYiCm91dHB1dDogIAogIGh0bWxfbm90ZWJvb2s6CiAgICBjb2RlX2ZvbGRpbmc6IHNob3cKICAgIGhpZ2hsaWdodDogZGVmYXVsdAogICAgI2tlZXBfbWQ6IHllcwogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMKICAgIHRoZW1lOiBqb3VybmFsCiAgICB0b2M6IHllcwogICAgdG9jX2Zsb2F0OgogICAgICBjb2xsYXBzZWQ6IG5vCiAgICAgIHNtb290aF9zY3JvbGw6IHllcwogICAgICB0b2NfZGVwdGg6IDMKY3NsOiByZWZlcmVuY2VzL2lzbWUuY3NsCmJpYmxpb2dyYXBoeTogcmVmZXJlbmNlcy9iaWJsaW9ncmFwaHkuYmliCmZvbnQtaW1wb3J0OiBodHRwOi8vZm9udHMuZ29vZ2xlYXBpcy5jb20vY3NzP2ZhbWlseT1SaXNxdWUKZm9udC1mYW1pbHk6ICdSaXNxdWUnCi0tLQoKPCEtLUJFR0lOOiAgU2V0IHRoZSBnbG9iYWwgb3B0aW9ucyBhbmQgbG9hZCBwYWNrYWdlcy0tPgpgYGB7ciBzZXQtZ2xvYmFsLW9wdGlvbnMsIGVjaG8gPSBGQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGV2YWwgPSBUUlVFLCAKICAgICAgICAgICAgICAgICAgICAgIGVjaG8gPSBUUlVFLCAKICAgICAgICAgICAgICAgICAgICAgIGNhY2hlID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgIGNvbGxhcHNlID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICBkZXBlbmRzb24gPSBOVUxMLAogICAgICAgICAgICAgICAgICAgICAgZW5naW5lID0gIlIiLCAjIENodW5rcyB3aWxsIGFsd2F5cyBoYXZlIFIgY29kZSwgdW5sZXNzIG5vdGVkCiAgICAgICAgICAgICAgICAgICAgICBlcnJvciA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICBmaWcucGF0aD0iRmlndXJlcy8iLCAgIyBTZXQgdGhlIGZpZ3VyZSBvcHRpb25zCiAgICAgICAgICAgICAgICAgICAgICBmaWcuYWxpZ24gPSAiY2VudGVyIiwgCiAgICAgICAgICAgICAgICAgICAgICBmaWcud2lkdGggPSA3LAogICAgICAgICAgICAgICAgICAgICAgZmlnLmhlaWdodCA9IDcpCmBgYAoKI0ludHJvZHVjdGlvbgoxLiAqKlRNTEUqKiBpcyBhIGdlbmVyYWwgYWxnb3JpdGhtIGZvciB0aGUgY29uc3RydWN0aW9uIG9mIGRvdWJsZSByb2J1c3QsIHNlbWlwYXJhbWV0cmljLCBlZmZpY2llbnQgc3Vic3RpdHV0aW9uIGVzdGltYXRvcnMuIFRNTEUgYWxsb3dzIGZvciBkYXRhLWFkYXB0aXZlIGVzdGltYXRpb24gd2hpbGUgb2J0YWluaW5nIHZhbGlkIHN0YXRpc3RpY2FsIGluZmVyZW5jZS4gCjIuIEFsdGhvdWdoICoqVE1MRSoqIGltcGxlbXRhdGlvbiB1c2VzIHRoZSBHLWNvbXB1dGF0aW9uIGVzdGltYW5kIChHLWZvcm11bGEpLiBCcmllZmx5LCB0aGUgVE1MRSBhbGdvcml0aG0gdXNlcyBpbmZvcm1hdGlvbiBpbiB0aGUgZXN0aW1hdGVkIGV4cG9zdXJlIG1lY2hhbmlzbSBQKEF8VykgdG8gdXBkYXRlIHRoZSBpbml0aWFsIGVzdGltYXRvciBvZiB0aGUgY29uZGl0aW9uYWwgbWVhbiBFJF97MH0kKFl8QSxXKS4gCjMuIFRoZSB0YXJnZXRlZCBlc3RpbWF0ZXMgYXJlIHRoZW4gc3Vic3RpdHV0ZWQgaW50byB0aGUgcGFyYW1ldGVyIG1hcHBpbmcuIFRoZSB1cGRhdGluZyBzdGVwIGFjaGlldmVzIGEgdGFyZ2V0ZWQgYmlhcyByZWR1Y3Rpb24gZm9yIHRoZSBwYXJhbWV0ZXIgb2YgaW50ZXJlc3QgJFxwc2koUF97MH0pJCAodGhlIHRydWUgdGFyZ2V0IHBhcmFtZXRlcikgYW5kIHNlcnZlcyB0byBzb2x2ZSB0aGUgZWZmaWNpZW50IHNjb3JlIGVxdWF0aW9uLiBBcyBhIHJlc3VsdCwgVE1MRSBpcyBhICoqZG91YmxlIHJvYnVzdCBlc3RpbWF0b3IqKi4KNC4gKipUTUxFKiogaXQgd2lsbCBiZSBjb25zaXN0ZW50IGZvciAkXHBzaShQX3swfSkkIGlzIGVpdGhlciB0aGUgY29uZGl0aW9uYWwgZXhwZWN0YXRpb24gRSRfezB9JChZfEEsVykgb3IgdGhlIGV4cG9zdXJlIG1lY2hhbmlzbSBQJF97MH0kKEF8VykgYXJlIGVzdGltYXRlZCBjb25zaXN0ZW50bHkuIFdoZW4gYm90aCBmdW5jdGlvbnMgYXJlIGNvbnNpc3RlbnRseSBlc3RpbWF0ZWQsIHRoZSAqKlRNTEUqKiB3aWxsIGJlIGVmZmljaWVudCBpbiB0aGF0IGl0IGFjaGlldmVzIHRoZSBsb3dlc3QgYXN5bXB0b3RpYyB2YXJpYW5jZSBhbW9uZyBhIGxhcmdlIGNsYXNzIG9mIGVzdGltYXRvcnMuIFRoZXNlIGFzeW1wdG90aWMgcHJvcGVydGllcyB0eXBpY2FsbHkgdHJhbnNsYXRlIGludG8gbG93ZXIgYmlhcyBhbmQgdmFyaWFuY2UgaW4gZmluaXRlIHNhbXBsZXMuW0BidWgyMDE2XQo1LiBUaGUgYWR2YW50YWdlcyBvZiBUTUxFIGhhdmUgYmVlbiByZXBlYXRlZGx5IGRlbW9uc3RyYXRlZCBpbiBib3RoIHNpbXVsYXRpb24gc3R1ZGllcyBhbmQgYXBwbGllZCBhbmFseXNlcy5bQHZhbjIwMTFdCjYuIFRoZSBwcm9jZWR1cmUgaXMgYXZhaWxhYmxlIHdpdGggc3RhbmRhcmQgc29mdHdhcmUgc3VjaCBhcyB0aGUgKip0bWxlKiogcGFja2FnZSBpbiBSIFtAZ3J1YmVyMjAxMV0uCgojVGhlIEctRm9ybXVsYQoxLiAkXHBzaShQX3swfSlcLD1cLFxzdW1fe3d9XCxcbGVmdFtcc3VtX3t5fVwsUChZPXlcbWlkIEE9MSxXPXcpLVwsXHN1bV97eX1cLFAoWSA9IHlcbWlkIEE9MCxXPXcpXHJpZ2h0XVAoVz13KSQgIAp3aGVyZSAgCiRQKFkgPSB5IFxtaWQgQSA9IGEsIFcgPSB3KVwsPVwsXGZyYWN7UChXID0gdywgQSA9IGEsIFkgPSB5KX17XHN1bV97eX1cLFAoVyA9IHcsIEEgPSBhLCBZID0geSl9JCAgCmlzIHRoZSBjb25kaXRpb25hbCBwcm9iYWJpbGl0eSBkaXN0cmlidXRpb24gb2YgWSA9IHksIGdpdmVuIEEgPSBhLCBXID0gdyBhbmQsICAKJFAoVyA9IHcpXCw9XCxcc3VtX3t5LGF9XCxQKFcgPSB3LCBBID0gYSwgWSA9IHkpJCAgCjIuIFVzaW5nIGNsYXNzaWNhbCByZWdyZXNzaW9uIG1ldGhvZHMgdG8gY29udHJvbCBjb25mb3VuZGluZyByZXF1aXJlcyBtYWtpbmcgdGhlIGFzc3VtcHRpb24gdGhhdCB0aGUgZWZmZWN0IG1lYXN1cmUgaXMgY29uc3RhbnQgYWNyb3NzIGxldmVscyBvZiBjb25mb3VuZGVycyBpbmNsdWRlZCBpbiB0aGUgbW9kZWwuCjMuIEFsdGVybmF0aXZlbHksICoqc3RhbmRhcmRpemF0aW9uKiogYWxsb3dzIHVzIHRvIG9idGFpbiBhbiB1bmNvbmZvdW5kZWQgc3VtbWFyeSBlZmZlY3QgbWVhc3VyZSB3aXRob3V0IHJlcXVpcmluZyB0aGlzIGFzc3VtcHRpb24uVGhlICoqRy1mb3JtdWxhKiogaXMgYSAqZ2VuZXJhbGl6YXRpb24gb2Ygc3RhbmRhcmRpemF0aW9uKltAcm9iaW5zMTk4Nl0KCiNDYXVzYWwgYXNzdW1wdGlvbnMgClVuZGVyIHRoZSBjb3VudGVyZmFjdHVhbCBmcmFtZXdvcmssIHdlIGhhdmUgdG8gY29uc2lkZXIgdGhlIGZvbGxvd2luZyBhc3N1bXB0aW9ucyB0byBjb25zaWRlciB0aGUgZXN0aW1hdGUgb2YgdGhlIEFURSBhcyBjYXVzYWw6IAoKIyNDTUkgb3IgUmFuZG9taXphdGlvbiAKKCRZX3swfSxZX3sxfVxwZXJwJEF8Vykgb2YgdGhlIGJpbmFyeSB0cmVhdG1lbnQgZWZmZWN0IChBKSBvbiB0aGUgb3V0Y29tZSAoWSkgZ2l2ZW4gdGhlIHNldCBvZiBvYnNlcnZlZCBjb3ZhcmlhdGVzIChXKSwgd2hlcmUgVyA9IChXMSwgIFcyLCBXMywg4oCmICwgV2spLiAKCiMjUG9zaXRpdml0eSAKYSDPtSBBOiBQKEE9YSB8IFcpID4gMCAgClAoQT0xfFc9dykgPiAwIGFuZCBQKEE9MHwgVyA9IHcpID4gMCBmb3IgZWFjaCBwb3NzaWJsZSB3LiAgCgojI0NvbnNpc3RlbmN5IG9yIFNVVFZBOiAKVGhlIG9ic2VydmVkIG91dGNvbWUgdmFsdWUsIHVuZGVyIHRoZSBvYnNlcnZlZCB0cmVhdG1lbnQsIGlzIGVxdWFsIHRvIHRoZSBjb3VudGVyZmFjdHVhbCBvdXRjb21lIGNvcnJlc3BvbmRpbmcgdG8gdGhlIG9ic2VydmVkIHRyZWF0bWVudCBmb3IgaWRlbnRpY2FsIGluZGVwZW5kZW50IGRpc3RyaWJ1dGVkIChpLmkuZC4pIHZhcmlhYmxlcy4gICAgCgojVE1MRSBmbG93IGNoYXJ0IAoqKlNvdXJjZSoqIDoJTWFyayB2YW4gZGVyIExhYW4gYW5kIFNoZXJyaSBSb3NlLiBUYXJnZXRlZCBsZWFybmluZzogY2F1c2FsIGluZmVyZW5jZSBmb3Igb2JzZXJ2YXRpb25hbCBhbmQgZXhwZXJpbWVudGFsIGRhdGFTcHJpbmdlciBTZXJpZXMgaW4gU3RhdGlzdGljcywgMjAxMQohW10oRmlndXJlcy90bWxlLnBuZykKCiNEYXRhIGdlbmVyYXRpb24KSW4gUiB3ZSBjcmVhdGUgYSBmdW5jdGlvbiB0byBnZW5lcmF0ZSB0aGUgZGF0YSB3aXRoIHRoZSBpbnB1dCBudW1iZXIgb2YgZHJhd3MgYW5kIHRoZSBvdXRwdXQgdGhlIG9ic2VydmVkIGRhdGEgKE9ic0RhdGEpIHBsdXMgdGhlIGNvdW50ZXJmYWN0dWFscyAoWTEsIFkwKS4gICAKVGhlIG9ic2VydmVkIGRhdGE6ICAKMS4gWTogbW9ydGFsaXR5IGJpbmFyeSBpbmRpY2F0b3IgKDEgZGVhdGgsIDAgYWxpdmUpIAoyLiBBOiBiaW5hcnkgdHJlYXRtZW50IGZvciBlbWVyZ2VuY3kgcHJlc2VudGF0aW9uIGF0IGNhbmNlciBkaWFnbm9zaXMgICgxIEVQLCAwIE5vbkVQKSAgICAKMy4gVzE6IEdlbmRlciAoMSBtYWxlOyAwIGZlbWFsZSkgIAo0LiBXMjogQWdlIGF0IGRpYWdub3NpcyAoMCA8NjU7IDEgPj02NSkgIAo0LiBXMzogQ2FuY2VyIFROTSBjbGFzc2lmaWNhdGlvbiAoc2NhbGUgZnJvbSAxIHRvIDQpICAKNS4gVzQ6IENvbW9yYmlkaXRpZXMgKHNjYWxlIGZyb20gMSB0byA1KSAgCgpgYGB7cn0KI2luc3RhbGwucGFja2FnZXMoImJyb29tIikKb3B0aW9ucyhkaWdpdHM9MykKZ2VuZXJhdGVEYXRhIDwtIGZ1bmN0aW9uKG4pewogIHcxIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9MC41KQogIHcyIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9MC42NSkKICB3MyA8LSByb3VuZChydW5pZihuLCBtaW49MCwgbWF4PTQpLCBkaWdpdHM9MykKICB3NCA8LSByb3VuZChydW5pZihuLCBtaW49MCwgbWF4PTUpLCBkaWdpdHM9MykKICBBICA8LSByYmlub20obiwgc2l6ZT0xLCBwcm9iPSBwbG9naXMoLTAuNCArIDAuMip3MiArIDAuMTUqdzMgKyAwLjIqdzQpKQogIFkgIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9IHBsb2dpcygtMSArIEEgLTAuMSp3MSArIDAuMyp3MiArIDAuMjUqdzMgKyAwLjIqdzQpKQogIAogICMgY291bnRlcmZhY3R1YWwKICBZLjEgPC0gcmJpbm9tKG4sIHNpemU9MSwgcHJvYj0gcGxvZ2lzKC0xICsgMSAtMC4xKncxICsgMC4zKncyICsgMC4yNSp3MyArIDAuMip3NCkpCiAgWS4wIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9IHBsb2dpcygtMSArIDAgLTAuMSp3MSArIDAuMyp3MiArIDAuMjUqdzMgKyAwLjIqdzQpKQogIAogICMgcmV0dXJuIGRhdGEuZnJhbWUKICBkYXRhLmZyYW1lKHcxLCB3MiwgdzMsIHc0LCBBLCBZLCBZLjEsIFkuMCkKfQpzZXQuc2VlZCg3ODU4KQpPYnNEYXRhIDwtIGdlbmVyYXRlRGF0YShuPTEwMDApClRydWVfUHNpIDwtIG1lYW4oT2JzRGF0YSRZLjEtT2JzRGF0YSRZLjApOwpjYXQoIiBUcnVlX1BzaToiLCBUcnVlX1BzaSkKQmlhc19Qc2kgPC0gbG0oZGF0YT1PYnNEYXRhLCBZfiBBKQpjYXQoIlxuIikKY2F0KCJcbiBOYWl2ZV9CaWFzZWRfUHNpOiIsc3VtbWFyeShCaWFzX1BzaSkkY29lZlsyLCAxXSkKYGBgCgojRGF0YSB2aXN1YWxpemF0aW9uCmBgYHtyfQojIERUIHRhYmxlID0gaW50ZXJhY3RpdmUKIyBpbnN0YWxsLnBhY2thZ2VzKCJEVCIpICMgaW5zdGFsbCBEVCBmaXJzdApsaWJyYXJ5KERUKQpkYXRhdGFibGUoaGVhZChPYnNEYXRhLCBuID0gbnJvdyhPYnNEYXRhKSksIG9wdGlvbnMgPSBsaXN0KHBhZ2VMZW5ndGggPSA1LCBkaWdpdHMgPSAyKSkKYGBgCgojVE1MRSBpbXBsZW1lbnRhdGlvbgoKIyMxc3Qgc3RlcDogRSRfezB9JChZfEEsVykgIApFc3RpbWF0aW9uIG9mIHRoZSBpbml0aWFsIHByb2JhYmlsaXR5IG9mIHRoZSBvdXRjb21lIChZKSBnaXZlbiB0aGUgdHJlYXRtZW50IChBKSBhbmQgdGhlIHNldCBvZiBjb3ZhcmlhdGVzIChXKSwgZGVub3RlZCBhcyB0aGUgJFFfezB9JChBLCoqVyoqKS4gVG8gZXN0aW1hdGUgJFFfezB9JChBLCoqVyoqKSB3ZSBjYW4gdXNlIGEgc3RhbmRhcmQgbG9naXN0aWMgcmVncmVzc2lvbiBtb2RlbDogCgokbG9naXRbUChZPTF8QSxXKV1cLD1cLFxiZXRhX3swfVwsK1wsXGJldGFfezF9QVwsK1wsXGJldGFfezJ9XntUfSQqKlcqKi4gICAgCgpUaGVyZWZvcmUsIHdlIGNhbiBlc3RpbWF0ZSB0aGUgaW5pdGlhbCBwcm9iYWJpbGl0eSAoYXMgZm9sbG93czogIC4gICAgICAoMSkKVGhlIHByZWRpY3RlZCBwcm9iYWJpbGl0eSBjYW4gYmUgZXN0aW1hdGVkIHVzaW5nIHRoZSBTdXBlciBMZWFybmVyIGxpYnJhcnkgaW1wbGVtZW50ZWQgaW4gdGhlIFIgcGFja2FnZSDigJxTdXBlci1MZWFybmVy4oCdNiB0byBpbmNsdWRlIGFueSB0ZXJtcyB0aGF0IGFyZSBmdW5jdGlvbnMgb2YgQSBvciBXIChlLmcuLCBwb2x5bm9taWFsIHRlcm1zIG9mIEEgYW5kIFcsIGFzIHdlbGwgYXMgdGhlIGludGVyYWN0aW9uIHRlcm1zIG9mIEEgYW5kIFcsIGNhbiBiZSBjb25zaWRlcmVkKS4gQ29uc2VxdWVudGx5LCBmb3IgZWFjaCBzdWJqZWN0LCB0aGUgcHJlZGljdGVkIHByb2JhYmlsaXRpZXMgZm9yIGJvdGggcG90ZW50aWFsIG91dGNvbWVzICBhbmQgIGNhbiBiZSBlc3RpbWF0ZWQgYnkgc2V0dGluZyBBID0gMCBhbmQgQSA9IDEgZm9yIGV2ZXJ5b25lIHJlc3BlY3RpdmVseToKIGFuZCwuCgoKCiMgVGhhbmsgeW91ICAKVGhhbmsgeW91IGZvciBwYXJ0aWNpcGF0aW5nIGluIHRoaXMgdHV0b3JpYWwuICAKSWYgeW91IGhhdmUgdXBkYXRlcyBvciBjaGFuZ2VzIHRoYXQgeW91IHdvdWxkIGxpa2UgdG8gbWFrZSwgcGxlYXNlIHNlbmQgPGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL21pZ2FyaWFuZS9NQUxGIiB0YXJnZXQ9Il9ibGFuayI+bWU8L2E+IGEgcHVsbCByZXF1ZXN0LgpBbHRlcm5hdGl2ZWx5LCBpZiB5b3UgaGF2ZSBhbnkgcXVlc3Rpb25zLCBwbGVhc2UgZS1tYWlsIG1lLiAgCioqTWlndWVsIEFuZ2VsIEx1cXVlIEZlcm5hbmRleioqICAKKipFLW1haWw6KiogKm1pZ3VlbC1hbmdlbC5sdXF1ZSBhdCBsc2h0bS5hYy51ayogIAoqKlR3aXR0ZXIqKiBgQFdBVFpJTEVJYCAgCgojIFNlc3Npb24gSW5mbyAKYGBge3Igc2Vzc2lvbi1pbmZvLCByZXN1bHRzID0nbWFya3VwJ30KZGV2dG9vbHM6OnNlc3Npb25faW5mbygpCmBgYAojIFJlZmVyZW5jZXMgCg==